Investigating the Py-ART data model for radar data

Scott Collis1 and Jonathan Helmus1
1:Argonne National Laboratory

This notebook explores the data model used for pointing radar data in the Python ARM Radar Toolkit (Py-ART). We will do this by loading a radar file from CF-Radial from ARM's X-Band system in the North Slope of Alaska Barrow site.

If you are not running this notebook in the short course VM, go fetch the data from HERE and place it in a data directory in the same directory as this notebook.


In [1]:
import pyart
from matplotlib import pyplot as plt
%matplotlib inline
filename = 'data/nsaxsaprrhiC1.a1.20131203.141936.nc'

Read the CF-Radial file into Py-ART's data model for pointing gated data


In [2]:
radar = pyart.io.read(filename)

Lets investigate what is at the top level with a dir() command


In [3]:
dir(radar)


Out[3]:
['__class__',
 '__delattr__',
 '__dict__',
 '__doc__',
 '__format__',
 '__getattribute__',
 '__hash__',
 '__init__',
 '__module__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 '_dic_info',
 'add_field',
 'add_field_like',
 'altitude',
 'altitude_agl',
 'antenna_transition',
 'azimuth',
 'elevation',
 'extract_sweeps',
 'fields',
 'fixed_angle',
 'info',
 'instrument_parameters',
 'latitude',
 'longitude',
 'metadata',
 'ngates',
 'nrays',
 'nsweeps',
 'radar_calibration',
 'range',
 'scan_rate',
 'scan_type',
 'sweep_end_ray_index',
 'sweep_mode',
 'sweep_number',
 'sweep_start_ray_index',
 'target_scan_rate',
 'time']

Anything in the data model which contains array-like data is a dictionary with metadata and the actual data contained in the 'data' key, for example the array which contains information about the elevation angle of the sensor.


In [4]:
radar.azimuth.keys()


Out[4]:
[u'comment', u'long_name', u'standard_name', u'units', 'data', u'axis']

In [5]:
radar.azimuth['standard_name']


Out[5]:
u'beam_azimuth_angle'

In [6]:
radar.azimuth['data']


Out[6]:
array([  96.88842773,   96.88842773,   96.88842773, ...,  186.43798828,
        186.43798828,  186.43798828], dtype=float32)

In [7]:
f = plt.figure(figsize=[15,8])
plt.plot(radar.time['data'], radar.azimuth['data'] )
plt.xlabel(radar.time['standard_name'] + ' (' + radar.time['units'] + ')')
plt.ylabel(radar.azimuth['standard_name'] + ' (' + radar.azimuth['units'] + ')')


Out[7]:
<matplotlib.text.Text at 0x7f39948e1b90>

So all the pointing data is contained in the base object, the azimuth and elevation of the antenna/sensor plus the range and time axes


In [8]:
print radar.range['data'].min(), radar.range['data'].max(), radar.range['units']
f = plt.figure(figsize=[15,8])
plt.plot(radar.time['data'], radar.elevation['data'] )
plt.xlabel(radar.time['standard_name'] + ' (' + radar.time['units'] + ')')
plt.ylabel(radar.elevation['standard_name'] + ' (' + radar.elevation['units'] + ')')


0.0 39950.0 meters
Out[8]:
<matplotlib.text.Text at 0x7f399492ca10>

there is also a swag of metadata contained within, well.. the metadata dictionary


In [9]:
for mykey in radar.metadata.keys():
    print mykey, ': ', radar.metadata[mykey]


comment :  Data in this file has not be calibrated, corrected, or had any quality control performed, use with caution.
instrument_type :  radar
site_id :  nsa
references :  http://radar.arm.gov/ARM_Radars/XSAPR.html
volume_number :  0
process_version :  $
title :  Atmospheric Radiation Measurement (ARM) program X-band Scanning ARM Precipitation Radar (XSAPR) raw moments
command_line :  xsapr_ingest -s nsa -f C1
source :  Atmospheric Radiation Measurement (ARM) program X-band Scanning ARM Precipitation Radar (XSAPR)
instrument_name :  xsapr-nsar1
platform_is_mobile :  false
data_level :  a1
datastream :  nsaxsaprrhiC1.a1
facility_id :  C1
institution :  United States Department of Energy - Atmospheric Radiation Measurement (ARM) program
n_gates_vary :  false
primary_axis :  axis_z
dod_version :  xsaprrhi-a1-1.2
Conventions :  CF/Radial instrument_parameters
platform_id :  xsaprrhi
platform_type :  fixed
location_description :  North Slope of Alaska (NSA), Barrow, Alaska
input_source :  /data/home/dev/ingest/xsapr/DATA/data/collection/nsa/nsaxsaprC1.00/BRW131203141935.RAWWEHR
field_names :  total_power, reflectivity_horizontal, mean_doppler_velocity, doppler_spectrum_width, differential_reflectivity, specific_differential_phase, cross_correlation_ratio, normalized_coherent_power, differential_phase
original_container :  sigmet
history :  created by user jhelmus on machine jade at 2014-01-13 16:18:54, using $

Now the final top level bit of information, the data model we use follows CF-Radial morphology and hence has a set of "helper" fields to format out the radar coverage pattern. That is, to seperate sweeps.


In [10]:
radar.sweep_end_ray_index['data']
f = plt.figure(figsize=[15,8])
for i in range(len(radar.sweep_end_ray_index['data'])):
    start_index = radar.sweep_start_ray_index['data'][i]
    end_index = radar.sweep_end_ray_index['data'][i]
    plt.plot(radar.time['data'][start_index:end_index], 
             radar.elevation['data'][start_index:end_index], 
             label = 'Sweep number '+ str(radar.sweep_number['data'][i]))
plt.legend()
plt.xlabel(radar.time['standard_name'] + ' (' + radar.time['units'] + ')')
plt.ylabel(radar.elevation['standard_name'] + ' (' + radar.elevation['units'] + ')')


Out[10]:
<matplotlib.text.Text at 0x7f39946c3150>

Now to the actual data, or what ARM would call Primary Measurements. This is all stored in the field field of the radar object and is a dictionary of dictionaries. Best shown by example:


In [11]:
print radar.fields.keys()
print ""
for mykey in radar.fields.keys():
    print mykey,':', radar.fields[mykey]['standard_name'] + ' (' + radar.fields[mykey]['units'] + ')'


[u'reflectivity_horizontal', u'cross_correlation_ratio', u'normalized_coherent_power', u'total_power', u'mean_doppler_velocity', u'doppler_spectrum_width', u'differential_reflectivity', u'specific_differential_phase', u'differential_phase']

reflectivity_horizontal : equivalent_reflectivity_factor (dBZ)
cross_correlation_ratio : cross_correlation_ratio_hv (unitless)
normalized_coherent_power : normalized_coherent_power (unitless)
total_power : equivalent_reflectivity_factor (dBZ)
mean_doppler_velocity : radial_velocity_of_scatters_away_from_instrument (meters_per_second)
doppler_spectrum_width : doppler_spectrum_width (meters_per_second)
differential_reflectivity : log_differential_reflectivity_hv (dB)
specific_differential_phase : specific_differential_phase_hv (degrees_per_km)
differential_phase : differential_phase_hv (degrees)

As far as CF-Radial ingest and write is concerned the variable names correspond to the variable names, the non-array data to the variable attributes and the 'data' key to the array.. lets look at some data


In [12]:
f = plt.figure(figsize=[15,8])
my_pc = plt.pcolormesh(radar.range['data'], radar.time['data'],
                       radar.fields['reflectivity_horizontal']['data'])
plt.xlabel(radar.range['standard_name'] + ' (' + radar.range['units'] + ')')
plt.ylabel(radar.time['standard_name'] + ' (' + radar.time['units'] + ')')
cb = plt.colorbar(mappable = my_pc)
cb.set_label(radar.fields['reflectivity_horizontal']['standard_name'] +\
             ' (' + radar.fields['reflectivity_horizontal']['units'] + ')')


And of course we can use our sweep indicators to isolate a single sweep


In [13]:
f = plt.figure(figsize=[15,8])
start_index = radar.sweep_start_ray_index['data'][0]
end_index = radar.sweep_end_ray_index['data'][0]
my_pc = plt.pcolormesh(radar.range['data'], radar.time['data'][start_index:end_index],
                       radar.fields['reflectivity_horizontal']['data'][start_index:end_index, :])
plt.xlabel(radar.range['standard_name'] + ' (' + radar.range['units'] + ')')
plt.ylabel(radar.time['standard_name'] + ' (' + radar.time['units'] + ')')
cb = plt.colorbar(mappable = my_pc)
cb.set_label(radar.fields['reflectivity_horizontal']['standard_name'] +\
             ' (' + radar.fields['reflectivity_horizontal']['units'] + ')')


We can get a quick overview of what is contained in the radar object using the info method.


In [14]:
radar.info('compact')   # see what happens with 'standard' or 'full'


altitude: <ndarray of type: float64 and shape: (1,)>
altitude_agl: None
antenna_transition: None
azimuth: <ndarray of type: float32 and shape: (1196,)>
elevation: <ndarray of type: float32 and shape: (1196,)>
fields:
	reflectivity_horizontal: <ndarray of type: float32 and shape: (1196, 800)>
	cross_correlation_ratio: <ndarray of type: float32 and shape: (1196, 800)>
	normalized_coherent_power: <ndarray of type: float32 and shape: (1196, 800)>
	total_power: <ndarray of type: float32 and shape: (1196, 800)>
	mean_doppler_velocity: <ndarray of type: float32 and shape: (1196, 800)>
	doppler_spectrum_width: <ndarray of type: float32 and shape: (1196, 800)>
	differential_reflectivity: <ndarray of type: float32 and shape: (1196, 800)>
	specific_differential_phase: <ndarray of type: float32 and shape: (1196, 800)>
	differential_phase: <ndarray of type: float32 and shape: (1196, 800)>
fixed_angle: <ndarray of type: float32 and shape: (2,)>
instrument_parameters:
	prt: <ndarray of type: float32 and shape: (1196,)>
	unambiguous_range: <ndarray of type: float32 and shape: (1196,)>
	prt_mode: <ndarray of type: |S1 and shape: (2, 32)>
	nyquist_velocity: <ndarray of type: float32 and shape: (1196,)>
latitude: <ndarray of type: float64 and shape: (1,)>
longitude: <ndarray of type: float64 and shape: (1,)>
nsweeps: 2
ngates: 800
nrays: 1196
radar_calibration:
range: <ndarray of type: float32 and shape: (800,)>
scan_rate: None
scan_type: rhi
sweep_end_ray_index: <ndarray of type: int32 and shape: (2,)>
sweep_mode: <ndarray of type: |S1 and shape: (2, 32)>
sweep_number: <ndarray of type: int32 and shape: (2,)>
sweep_start_ray_index: <ndarray of type: int32 and shape: (2,)>
target_scan_rate: None
time: <ndarray of type: float64 and shape: (1196,)>
metadata:
	comment: Data in this file has not be calibrated, corrected, or had any quality control performed, use with caution.
	instrument_type: radar
	site_id: nsa
	references: http://radar.arm.gov/ARM_Radars/XSAPR.html
	volume_number: 0
	process_version: $
	title: Atmospheric Radiation Measurement (ARM) program X-band Scanning ARM Precipitation Radar (XSAPR) raw moments
	command_line: xsapr_ingest -s nsa -f C1
	source: Atmospheric Radiation Measurement (ARM) program X-band Scanning ARM Precipitation Radar (XSAPR)
	instrument_name: xsapr-nsar1
	platform_is_mobile: false
	data_level: a1
	datastream: nsaxsaprrhiC1.a1
	facility_id: C1
	institution: United States Department of Energy - Atmospheric Radiation Measurement (ARM) program
	n_gates_vary: false
	primary_axis: axis_z
	dod_version: xsaprrhi-a1-1.2
	Conventions: CF/Radial instrument_parameters
	platform_id: xsaprrhi
	platform_type: fixed
	location_description: North Slope of Alaska (NSA), Barrow, Alaska
	input_source: /data/home/dev/ingest/xsapr/DATA/data/collection/nsa/nsaxsaprC1.00/BRW131203141935.RAWWEHR
	field_names: total_power, reflectivity_horizontal, mean_doppler_velocity, doppler_spectrum_width, differential_reflectivity, specific_differential_phase, cross_correlation_ratio, normalized_coherent_power, differential_phase
	original_container: sigmet
	history: created by user jhelmus on machine jade at 2014-01-13 16:18:54, using $

This functionality is also available from the command line using the radar_info command.


In [15]:
!radar_info --compact data/nsaxsaprrhiC1.a1.20131203.141936.nc


altitude: <ndarray of type: float64 and shape: (1,)>
altitude_agl: None
antenna_transition: None
azimuth: <ndarray of type: float32 and shape: (1196,)>
elevation: <ndarray of type: float32 and shape: (1196,)>
fields:
	reflectivity_horizontal: <ndarray of type: float32 and shape: (1196, 800)>
	cross_correlation_ratio: <ndarray of type: float32 and shape: (1196, 800)>
	normalized_coherent_power: <ndarray of type: float32 and shape: (1196, 800)>
	total_power: <ndarray of type: float32 and shape: (1196, 800)>
	mean_doppler_velocity: <ndarray of type: float32 and shape: (1196, 800)>
	doppler_spectrum_width: <ndarray of type: float32 and shape: (1196, 800)>
	differential_reflectivity: <ndarray of type: float32 and shape: (1196, 800)>
	specific_differential_phase: <ndarray of type: float32 and shape: (1196, 800)>
	differential_phase: <ndarray of type: float32 and shape: (1196, 800)>
fixed_angle: <ndarray of type: float32 and shape: (2,)>
instrument_parameters:
	prt: <ndarray of type: float32 and shape: (1196,)>
	unambiguous_range: <ndarray of type: float32 and shape: (1196,)>
	prt_mode: <ndarray of type: |S1 and shape: (2, 32)>
	nyquist_velocity: <ndarray of type: float32 and shape: (1196,)>
latitude: <ndarray of type: float64 and shape: (1,)>
longitude: <ndarray of type: float64 and shape: (1,)>
nsweeps: 2
ngates: 800
nrays: 1196
radar_calibration:
range: <ndarray of type: float32 and shape: (800,)>
scan_rate: None
scan_type: rhi
sweep_end_ray_index: <ndarray of type: int32 and shape: (2,)>
sweep_mode: <ndarray of type: |S1 and shape: (2, 32)>
sweep_number: <ndarray of type: int32 and shape: (2,)>
sweep_start_ray_index: <ndarray of type: int32 and shape: (2,)>
target_scan_rate: None
time: <ndarray of type: float64 and shape: (1196,)>
metadata:
	comment: Data in this file has not be calibrated, corrected, or had any quality control performed, use with caution.
	instrument_type: radar
	site_id: nsa
	references: http://radar.arm.gov/ARM_Radars/XSAPR.html
	volume_number: 0
	process_version: $
	title: Atmospheric Radiation Measurement (ARM) program X-band Scanning ARM Precipitation Radar (XSAPR) raw moments
	command_line: xsapr_ingest -s nsa -f C1
	source: Atmospheric Radiation Measurement (ARM) program X-band Scanning ARM Precipitation Radar (XSAPR)
	instrument_name: xsapr-nsar1
	platform_is_mobile: false
	data_level: a1
	datastream: nsaxsaprrhiC1.a1
	facility_id: C1
	institution: United States Department of Energy - Atmospheric Radiation Measurement (ARM) program
	n_gates_vary: false
	primary_axis: axis_z
	dod_version: xsaprrhi-a1-1.2
	Conventions: CF/Radial instrument_parameters
	platform_id: xsaprrhi
	platform_type: fixed
	location_description: North Slope of Alaska (NSA), Barrow, Alaska
	input_source: /data/home/dev/ingest/xsapr/DATA/data/collection/nsa/nsaxsaprC1.00/BRW131203141935.RAWWEHR
	field_names: total_power, reflectivity_horizontal, mean_doppler_velocity, doppler_spectrum_width, differential_reflectivity, specific_differential_phase, cross_correlation_ratio, normalized_coherent_power, differential_phase
	original_container: sigmet
	history: created by user jhelmus on machine jade at 2014-01-13 16:18:54, using $

Thus concludes the intro! Py-ART, of course, can do all this for you including pretty PPIs etc.. but this gives an introduction to the data model we use. Questions? Comments? Science Lead: Scott Collis Development lead: Jonathan Helmus.